Papers with auxiliary loss functions
Non-contrastive sentence representations via self-supervision (2024.findings-naacl)
Copied to clipboard
| Challenge: | Text embeddings are an important tool for a variety of NLP tasks. |
| Approach: | They compare sample contrastive methods with the standard baseline for contrastive sentence embeddings, SimCSE, and a class of self-supervised non-contrastive loss functions and methods. |
| Outcome: | The proposed methods outperform the standard baseline for contrastive sentence embeddings, SimCSE, on downstream tasks without auxiliary loss functions. |
BERTese: Learning to Speak to BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work shows that pre-trained language models encode large amounts of world knowledge in their parameters. |
| Approach: | They propose a method for automatically rewriting queries into a paraphrase query called "BERTese" they add auxiliary loss functions that encourage the query to correspond to actual language tokens . |
| Outcome: | The proposed method outperforms baselines and provides some insight into the type of language that helps language models perform knowledge extraction. |
Exploring Distantly-Labeled Rationales in Neural Network Models (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods focus on distantly-labeled rationales, ignoring the potential important non-rationale words and not distinguishing the importance of different rationale words. |
| Approach: | They propose two novel auxiliary loss functions to make better use of distantly-labeled rationales, which encourage models to maintain their focus on important words beyond labeled rationals (PINs) and alleviate redundant training on non-helpful rationale (NoIRs). |
| Outcome: | The proposed methods outperform existing methods on two representative classification tasks while maintaining the ability to spread focus to other unlabeled important words. |